Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TB Pipeline on Amazon Genomics CLI #24

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open

TB Pipeline on Amazon Genomics CLI #24

wants to merge 60 commits into from

Conversation

mmueller76
Copy link

Changes to enable workflow execution on Amazon Genomics CLI (AGC):

  • agc-project.yml added
  • script provision_resources.sh to provision KrakenDB files, Bowtie index files and TB Pipeline resources on Amazon S3
  • Python scripts under bin/ made executable and update of script path in workflow modules
  • input channel for singularity and resources folders defined
  • input.json template added (modify as required)
  • MANIFEST.json file added
  • dryrun-test-agc.py script to execute dry runs on AGC added
  • scripts docker/pull_and_push.sh and docker/build_and_push.sh to make Docker images available on Amazon Elastic Container Registry added
  • agc profile added to nextflow.config
  • report, timeline, trace and DAG generation configuration added to nextflow.config
  • Amazon Genomics CLI section added to README.md
  • --resources_dir, --report_dir and --container_registry workflow parameters added

mmueller76 and others added 30 commits September 26, 2022 16:20
@mmueller76
agc-project.yaml added
72066c3
@mmueller76
AGC context name changed
ee7d1f2
@mmueller76
agc profile added to nextflow.config
fb1dd53
@mmueller76
AGC MANIFEST and inputs.json for test scenario1 added
f2dd0e7
@mmueller76
test manifest and input.json moved to test/agc
28006bf
@mmueller76
input.json added and MANIFEST.json modified for full workflow run
4fdad13
@omrkha
python script to generate test scenario is being added
e7412e3
@mmueller76
fixed input file path bug in software-json.py
5dd02c7
@mmueller76
make singularity directory a workflow input
324b66f
@mmueller76
make resources worklow input
7a4f76f
@mmueller76
white space change
50de0d0
@mmueller76
remove resources from project
5de881a
@mmueller76
tests-scenario-script.py removed
f3442e4
@mmueller76
add build_and_push.sh script to build Docker containers and push them…
6952490 
… to ECR
@mmueller76
made resources input of clockwork workflow
008145d
@mmueller76
pull_and-push.sh script to push Docker containers to ECR added
e501286
@mmueller76
fix binary name in gnomon process
4dbff96
@mmueller76
use Docker containers in ECR
e8fe6a7
@mmueller76
script to upload resources (bowtie index, KrakenDB and resouces conta…
6f3a482 
…ined in project to S3
@mmueller76
agc added to profile section of help message
14b4199
@mmueller76
defaultCtx name changed to ondemand
fa36883
@mmueller76
tag construction in build_and_push.sh refactored
9f36818
@mmueller76
report, trace, timeline and dag generation added to nextflow.config; …
793f243 
…resources_dir parameter removed from nextflow.config; public preporcessing container replaced with ECR version
@mmueller76
output_dir param removed from testing.config
df9f7ad
@mmueller76
fixed gnomonicus JSON output file name
b0dc499
@mmueller76
changed configuration of report, timeline, trace and dag generation
5600c84
@mmueller76
updated script to provision resources
b236cfd
@mmueller76
kraken_db and bowtie2_index parameters in testing.config updated
bac7e3d
@omrkha
script to submit test workflow runs
b5df9ba
@omrkha
script to upload resources to s3 finalised
fa0bbb2
@mmueller76
session ID added to runtime report
36550b7
@mmueller76
typo in context name in agc-project.yaml fixed
99be17a
@mmueller76
workflow.sessionId added to output directory path
638c1f0
@mmueller76
defaultCtx removed from AGC project YAML
3ccdcab
@mmueller76
tidy up dryrun-test-agc script
83b7506
@mmueller76
renamed ash-script-workflow.sh -> dryrun-test-agc.sh
b469dd5
@mmueller76
fixed indentation in main.nf
ee033ae
@mmueller76
cleaned up provision_resources.sh script
08d5c99
@mmueller76
merge conflict in modules/vcfpredictModules.nf resolved
1fb0696
@mmueller76
made python scripts executable and updated script path in modules
8a76832
@mmueller76
make Nextflow execution report path configurable
7e296b0
@mmueller76
deleted tests/agc/MANIFEST.scenario1.json
cc70ebf
@mmueller76
refactored tests/agc/generate-test-scenario-inputs.py
0c146ce
@mmueller76
consolidated tests/agc/generate-test-scenario-inputs.py and dryrun-te…
f9c271e 
…st-agc.sh into dryrun-test-agc.py
@mmueller76
resources directory add back into project directory
e68eedd
@mmueller76
workflow sessionId removed from output path
06e061c
@mmueller76
worklfow run submission command in dry run script fixed
953c7ad
@mmueller76
scenarios uncommented
dbfa697
@mmueller76
resources_dir and container_registry params added to nextflow.config
99e602f
@mmueller76
container_registry param added to inputs.json
7503bf4
@mmueller76
resources_dir and container_registry added to main.nf
3561f7c
@mmueller76
refactored provision_resources.sh script
82cc05a
@mmueller76
data section removed from agc-project.yaml file
3e6a825
@mmueller76
AGC section added to README.md
3e9104f
@mmueller76
inputs.json templated
2c78db1
@mmueller76
changes in testing.config reverted
6dd9483
@mmueller76
kraken_db and bowtie2_index params add to dryrun-test-agc.py
8f633a9
@mmueller76
output_dir reinstated in testing.config
f19ec2b
@mmueller76
white space change testing.config
2f03654
@annacprice
Copy link
Member

Hi @mmueller76, sorry for the delay in getting to this. We have just released a major update to the pipeline. You may wish to update your pull request, in particular to include Afanc which has replaced Mykrobe for speciation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mmueller76 @annacprice @omrkha